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Abstract - There is a growing need for humans to perform complex remote operations, and to extend the 
intelligence and experience of experts to distant applications. We assert that a blending of human intelligence, 
modern information technology, remote control and intelligent autonomous systems is required, and have coined 
the term “tele-autonomous technology,” or “tele-automation” for short, for methods for producing intelligent 
action at a distance. Tele-automation goes beyond autonomous control in that it blends in human intelligence 
and action as appropriate. It goes beyond tele-operation in that it incorporates as much autonomy as is 
possible/reasonable. We discuss in detail a new approach for solving one of the fundamental problems facing 
tele-autonomous systems: the need to overcome time delays due to telemetry and signal propagation. We 
introduce new concepts, called time and position clutches, that allow the time and position frames, respectively, 
between the local user control and the remote device being controlled, to be desynchronized. The design and 
implementation of these mechanisms are described in detail. We demonstrate that these mechanisms lead to 
substantial telemanipulation performance improvements, including the novel result of improvements even in 
the absence of time delays. The new controls also yield a simple protocol for handoffs of control of manipulation 
tasks between local operators and remote systems. 


1 Introduction 

There is a growing need for humans to be able to perform complex, large scale, remore opera- 
tions, and extend the intelligence and experience of experts to distant applications. This need is, 
perhaps, most dramatic in the area of space exploration. The National Commission on Space report 
Pioneering the Space Frontier [NAS86] describes in a vivid and exciting way the many potential 
scientific, commercial and colonization activities that could be accomplished in space over the next 
50 years, and the Ride report Leadership and America ’s Future in Space [RID87] discusses specific 
missions that could be adopted to lead the nation into the new space era. These missions will i en- 
quire vastly more complex and larger scale remote operations than previous missions. There is an 
equally strong need for this capability, though not as dramatically obvious, for terrestrial applica- 
tions. For example, undersea operations, mining, public safety, nuclear power maintenance, various 
defense applications and a wide variety of other hazardous operations would benefit strongly from an 
increased capability to perform complex remote operations. In this paper, we explore a new approach 
to achieving this type of operation. 

1.1 Dimensions of the Problem Space 

The terms “robotics”, “tele- presence”, “artificial intelligence” and “expert systems” appear 
throughout Pioneering the Space Fjvntier [NAS86] and the Ride report [RID87], and are indications 
of recognition of the need for complex, large scale remote operations. Yet, none of these technologies, 
either alone or in combination, are sufficient to satisfy the need for remote intelligent action. Robots, 
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artificial intelligence and expert systems are essential for building autonomous systems, while tele- 
operation and tele-presense are directed toward making the sensory-motor coordination of a remote 
robot lie entirely within the brain of a human operator. Pure tele-operation, however, is too awkward 
and human intensive to be fully effective. Communication time delays to/from remote operations 
further complicate the control. On the other hand, while AI technology is capable of fairly intelli- 
gent cognitive activities, it lacks the capability to manage most real-time manipulation tasks. Such 
approaches also do not adequately utilize the human intelligence that may be present at the remote 
site. 

Many opportunities have been overlooked by concentrating on total automation, artificial intelli- 
gence, or simple tele-manipulation alone. New, more generalized thinking is required. 

We assert that one must return to the fundamental problem space, and examine all possible cross 
products of its dimensions to determine new, more effective solutions. The sketch of Figure 1 shows 
our view of this basic problem space. An element of intelligent activity is an entity at a specific 
location performing a process (perception, cognition, or action), represented as a point in this space. 
Intelligent activity is a symphony of these elements connected by processes of communication, much 
the same way that language is formed of the elements of grammar and the rules governing their 
use. The space is discrete, and a single entity may be represented by more than one point (but all 
on a single location plane) if it performs more than one process. These dimensions imply a wide 
range of systems that are possible. Many kinds of devices, not just robots and vehicles, are possible. 
Manned operations are included. Perception, cognition and action may be divided or shared among 
entities. Cooperating entities may be at different locations. Cooperative relations among the entities 
may be dynamic. For example, tele-control of a device may be handed off between different humans 
in different locations. And, groups of humans and machines at different locations may be able to 
dynamically cooperate. For example, in one future scenario, humans at scattered locations - e.g., on 
Earth and in the Space Station - may cooperate on a repair satellite task using remote robots, as 
sketched in Figure 2. 

These dimensions also help place the traditional approaches in perspective. For example, a stan- 
dard tele-manipulation system would have two points in the human plane indicating perception and 
cognition, and a single point in the manipulator plane indicating remote action. An autonomous 
system would be a set of three points in the plane of some device. The traditional approaches are 
thus seen as constrained to a set of three points lying in one or two planes perpendicular to the 
“form” axis. Tele- robotics relaxes the constraints by allowing both the human and robot plane to 
have points of the same process type, indicating a sharing of the process, most commonly cognition. 
However, these improvements from tele-robotics are still quite limited in comparison to the full space 
of possibilities. 

We suggest that to achieve the most useful control configurations, a blending of modern informa- 
tion technology, human intelligence, remote control, and intelligent autonomous systems is required. 
When we first began work in this area, we coined the term “tele- autonomous technology” [CON87a, 
CON87b], or “tele- automation” for short, for the new methods for producing intelligent action at a 
distance, in order to emphasize the interactions of humans with remote, intelligent, partly-autonomous 
systems. We also envision tele-autonomous activities as typically involving several humans and several 
partly autonomous systems in coordinated activities. Thus the new tele-autonomous system technol- 
ogy will also blend in the methods of “collaboration technology” or “computer supported cooperative 
work” for effective goal-seeking coordination in such multi-agent systems [GRE88]. 

Tele-automation represents a new way of viewing and developing systems that must perform 
intelligent action at a distance. The remote systems will be as intelligent and autonomous as possi- 
ble/appropriate, but capable of being guided when necessary by humans. This can greatly reduce the 
need for continual human involvement, while complementing the powers of AI autonomous systems. 
Machine learning can provide the capability for the autonomous part of the system to gradually take 
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over more and more of the operation time the human generates inputs to the system. Collaboration 
technology can provide the ability for cooperative interactions. 

Of course, the intelligent agents at the remote location need not be autonomous devices, but 
might well be intelligent, but non-expert, humans. Many operations, such as infrequent maintenance 
operations or actions ensuing from unforeseen events, will be difficult or impossible to automate. 
Tele-autonomous technology will allow experts in one location to guide non-experts in another in 
much more effective ways than currently possible. The payoff in terms of reduced training costs 
(many astronauts trained for months to perform a 45 minute repair on Solar Max [ESS85]) and the 
extension of operations that can be performed in space will be tremendous. 


1.2 A New General Infrastructure for Remote Operations 

The future NASA missions described in the Commission on Space and the Ride reports depend 
heavily upon performing extensive, labor intensive intelligent actions in space. Examples are explo- 
ration, remote sensing, surveying, prospecting, mining, manufacturing, assembly, payload mainte- 
nance and servicing, agriculture, experimentation, and many routine daily operations. Details of new 
requirements for such remote systems are found, for example, in NASA’s Automation and Robotics 
Progress Reports [NAS88]. It would be extremely costly if humans were to directly perform all these 
operations. Even if these costs could be borne, there are real constraints on the number of humans 
that it will be possible to sustain in space for the next several decades, thus limiting both the amount 
of labor and expertise that can be directly resident. 

In the past, the research community has proposed “solutions through automation”. However, in 
recent years it has become clear that automation offers only a partial solution because of its inability 
to function robustly when dealing with spontaneous, unexpected events. While artificial intelligence 
promises to ease some of these “automation difficulties,” we note that the advances of AI have been 
primarily in the area of machine cognition. AI does not yet offer comparable advances in the area of 
machine perception and action. While current and near future technology offer us intelligent thinking 
machines,” they offer us only very limited means for providing “clever perceiving and manipulating 
machines.” 

The whole future of space exploration thus depends upon the creation of some form of infrastruc- 
ture for bringing a mixture of human and machine perception, thinking and manipulation skills to 
bear on the many tasks to be done, even though those skills may be scattered over large distances 
in space and time. While less obvious because the cost of human labor is not so high, the same 
infrastructure can have enormous payoff for many of the same applications in a terrestrial setting as 
well, particularly those applications involving hazardous or difficult to reach environments. Rather 
than foreseeing complete automation solutions to these problems, we instead visualize construction 
of an infrastructure that enables humans to collaboratively project and focus their capabilities to 
perform distant tasks. 

Of course, it will be important to provide the means to evolve toward more fully autonomous 
systems where that is feasible. We visualize the provision of new forms of machine learning technology 
that can mimic and learn oft repeated skills, thus relieving humans from having to perform tasks that 
have become somewhat routine. 

But the key point is that a new form of “work infrastructure” is essential to the rapid and efficient 
exploration and development of space, and will be highly useful to many terrestrial applications as 
well. We believe that tele-autonomous system technology is the basis for that new infrastructure. 


2 Overview of Tele-autonomous Operation 

Tele- automation represents a new way of viewing and developing systems that must perform intel- 
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ligent actions at a distance. It goes beyond autonomous control in that it blends in human intelligence 
and action as appropriate. Tele-automation goes beyond tele-operation in that it incorporates as much 
autonomy as is possible/reasonable. Tele- automation is also concerned with enabling collaboration 
among multiple human and autonomous systems, and with enabling adaptation, or machine learning, 
by and among the autonomous systems. 

Figure 3 shows, at a conceptual level, the structure of a single local/remote pair in a basic tele- 
autonomous system. The spatial reference frame is taken to be that of the human controller at the left 
side of the figure, i.e, the controlled environment is remote. The controlled environment can include 
humans and/or any manner of device. The remote intelligent controller receives data from multiple 
sensors, and provides multiple outputs, encompassing anything from servo level control signals to a 
robot joint, to video signals to a heads-up display worn by a remote human. 

The inputs on the local side of the system may be any form of input control by the human, from 
simple joystick control, to complex cockpits with many inputs, to discrete commands for the remote 
controller to perform complex tasks. The local display represents any kind of feedback to the human 
about the remote environment. This will include both simulated information and actual feedback 
signals and may be composed of TV images, complex graphics, force reflection on input devices, 
or even high speed data analysis. The distance between the local and remote sites can produce 
substantial time delays in the signal transmission between them. 

Tele-autonomous control of even a single local/remote controller pair provides many operating 
modes, including: 

1. Direct continuous tele-operator control of a remote device. The remote controller merely follows 
its inputs. This is currently the most common form of operation. 

2. Shared continuous tele-operator control of a remote device. The remote controller performs 
higher than position serving. For example, it might treat received inputs as being relative to 
an object to be manipulated and perform appropriate transformations before following them 
[VOL88]. Or, it might treat received inputs as a nominal path, and perform some local sensing 
and replanning to reach the goals of the nominal plan. 

3. Discrete command control by the human operator of the remote device. This implies a higher 
level of capability in the remote portion of the controller that can vary from simple setpoint 
control of a number of satellite antenna positioning servos, to complex task analysis, planning 
and execution. At this level the commands become highly task specific, though the lower level 
primitives utilized may be more generic. 

4. Supervisory control 3 . The remote device operates in a largely autonomous mode and only in- 
teracts with the human when it encounters a situation it cannot handle, i.e., management by 
exception, or in which the human notices an opportunity to improve performance, i.e, oppor- 
tunistic management. It differs from the discrete command mode principally in the frequency 
of interaction with the human controller, and the philosophy of being largely autonomous. One 
local human operator might supervise a fleet of remote devices. 

5. Learning control. The remote controller is given an intelligence that allows it to learn from 
human inputs and sensor information, and subsequently deduce correct behavior in similar 
situations without human intervention. 

6. Guidance of remote non-expert humans by local experts. In this mode a variety of media, visual 
displays, graphics, touching, pointing, etc., are used to achieve a collaboration between the local 
expert and the remote non-expert. 

3 We use the term supervisory control to describe a much higher level mode than that usually attributed to the term. 
However, our usage fits the intuitive interpretation if the term quite well. 
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Groups of such basic systems, possibly with local controllers in different locations, will make up larger 
scale tele-autonomous systems. Many kinds of interactions will be possible, from hand-offs of control 
between different local control agents (even if in different physical locations) to shared cooperative 
action of the remote devices. 


3 Basic Tele-automation Controls 

Fully general tele-autonomous systems do not yet exist, and will be the subject of research for a 
long time. However, we have recently discovered some fundamental principles that we believe will be 
part of the architectural foundation of almost any general tele-autonomous system. 

One of the most fundamental problems facing tele-autonomous systems is time delay due to 
telemetry and/or signal propagation delays. Even modest time delays have long been known to cause 
instabilities in control systems such as robots. And, the time delays present in space applications are 
anything but modest. 

We review here a sequence of interface control concepts originally presented in [CON87a] that col- 
lectively underlie efficient control of manipulation tasks and also enable simple protocols for exchange 
of such tasks among control agents. 

3.1 Coping with Time Delay 

Although tele-manipulation has been studied for years (e.g., see [GOE52], [KUG72], [HIL79], 
[DRA87], [MOL87]), Noyes and Sheridan [NOY84] were the first to make significant progress on the 
tele-manipulation time-delay problem. Noyes and Sheridan suggested that the operator control a 
local simulation of the telerobot, with the control signals then sent in parallel to the simulation and 
the remote telerobot. The simulation is then displayed superimposed over the return video. In this 
way the operator can “see” the effects of the control immediately without having to fully wait for 
the return signal from the telerobot. This system concept is sketched in Figure 4. In the system we 
built to test the concept, we used a model of the telerobot on the IRIS workstation, making it easy to 
simulate time delays and easing solution of the correspondence problem between the simulated and 
actual robots. 

Figure 5 presents a visualization of telerobotic manipulation using a forward simulation to cope 
with the time delay. The wire frame is the forward simulation that directly responds to operator 
control, and the solid frame represents the time delayed image of the real telerobot. Much faster 
and smoother control is achieved. Task time may be reduced to nearly that of the no-delay case, as 
shown in Figure 6. This is a first step towards evolving machine manipulation visualization, since 
the visualization could help cope not only with communication delays, but also with computational 
delays within a self-contained autonomous agent. 

3.2 The Time Clutch 

In the work of Noyes and Sheridan described above, the time frames of the simulation and the 
robot are separated by the time delay of the telemetry and propagation. However, there is no 
intrinsic reason to maintain this synchrony. We thus introduce the concept of a “time clutch that 
can disengage synchrony between operator specification time and telerobot manipulation time during 
path specification. Our hypothesis is that operators can generate a path faster than the robot can 
follow it. This is particularly true of large space telerobots such as the Remote Manipulator System 
(RMS) [NAS81]. Once generated, a path segment can then be followed more quickly by the robot 
than would be the case if the robot were time- synchronized to the specification process; with time 
synchrony disengaged, the robot can steadily proceed at nearly its maximum rate, subject of course 
to error limits and hard constraints. 
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Figure 7 shows a path being generated well out in advance of the actual robot by an operator 
using forward simulation with time clutch disengaged. The performance of an operator when using 
the time clutch while performing the task of touching a series of boxes in our experimental trials 
[CON87b] is shown in Figure 8. Remarkably, the performance is better than control without the time 
clutch even in the case of no time delay. 

This step in the evolution of machine manipulation visualization enables the cognitive agent to 
“look and think ahead” of the manipulation under control, with the look-ahead time being elastic, and 
not just a fixed internal or external system time delay. The implementation of this new capability 
requires only a simple mutation of the forward simulation previously used for coping with a time 
delay. 


3.3 The Position Clutch 

We next introduce the concept of a “position clutch” which enables a disengagement of position 
synchrony between simulator and manipulator path. We hypothesize that faster, shorter, cleaner 
paths can be generated on difficult tasks using this control. This idea is illustrated in Figure 9, which 
shows the use of the position clutch to disengage from path generation during a close approach to a 
difficult manipulation (in this case, touching a small object). 

Suppose, for example, that the operator had arrived (in the simulation) at point A ahead of time 
by using the time clutch. The position clutch can then be disengaged, stopping the output from the 
operator control from going to the real telerobot - it will only go to the simulation. When the forward 
simulator is in good position, the position clutch will be reengaged, causing a short, smooth path to 
be inserted that links to the earlier path. This avoids inclusion of jittery prepositioning movements 
in the final path to be followed. Further, the time spent by the operator in achieving the proper 
position will not be incurred by the real telerobot since these motions were “clipped” out of the path 
sent to the telerobot. 

The operator has thus used up some of the time saved through use of the time clutch, with the 
result that the overall task time of the telerobot is reduced still further. This level of manipulation 
visualization corresponds to quick visualizations and visualized trials of multiple alternatives prior 
to commitment to action, and its implementation requires only another simple mutation of the basic 
forward simulation capability. 

3.4 The Time Brake 

To handle contingencies and errors we introduce the concept of a time brake. This control can be 
used to deal with situations such as something falling over a previously generated path, as illustrated 
by the “X” in Figure 10. In Figure 10 we see the time brake being applied and the forward-simulated 
manipulator backing down the path (in a race to get on the other side of the obstacle before the real 
system gets there). 

This aspect of visualization corresponds to seeing something about to happen that will interrupt 
an action previously visualized but not yet underway. If it had gotten underway, or is allowed to get 
underway, the system will have to deal with it through local reflex action or crash. But, if visualized 
in time, the cognitive agent can withdraw the action using the time brake. 

3.5 Task HandofFs and Rendezvous 

These basic tele-autonomous system interface controls also provide the basis for a simple, elegant 
protocol for hand-offs and rendezvous of tasks between different control agents. Imagine two operators, 
one in control of the telerobot and the other about to take over in relief of the first, as sketched in 
Figure 11. Each operator would be in control of a simulation of the telerobot, but only the control 
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signals of the first would be sent to the real telerobot. The relief operator would, with position clutch 
disengaged, guide his/her simulation as close to the first operator’s as possible (or required). The 
first operator then disengages their position clutch, leaving the path “hanging”. Figure 12 shows this 
moment in the interaction. 

The second operator then engages their position clutch, rendezvousing with the path and taking 
control of future path generation. When the actual manipulator passes over this path segment, it will 
do so smoothly and will not notice that a change of control agent has occurred in mid-maneuver. We 
can again find interesting biological analogies to this visualization situation. For example, consider 
the interactions among basketball players as they previsualize fast-paced multiplayer interactions. 

We believe that this simple protocol can be built upon to mechanize quite a wide range of ma- 
nipulation interactions between autonomous agents. 

4 Future Directions 

Tele-autonomous technology presents new challenges in human computer interaction. We have 
proposed a set of interface controls that are conceptually simple and easy to mechanize. The con- 
trols are generic ones that may be applicable in many different specialized situations. They are 
also cognitively and manipulatively accessible to the uninitiated by analogy. But many other new 
human interface aspects haven’t been pinned down at all. How is the operator to visualize where 
they are, who has control of what, and who they give control to next as they enter or leave some 
subtask within a complex task lattice? What measures can we provide concerning operator perfor- 
mance, and what feedback can we provide? And what about the analysis and design of cognitive 
and manipulation tasks themselves? Research can perhaps provide better measures of joint human- 
machine cognitive-manipulative performance. Analyses similar to those in [CAR83] may then lead us 
to design intermixings of human and machine activity that yield substantial improvements in overall 
performance. 

The work poses some additional new challenges in robotics, such as the eventual need to perceive, 
model and forward simulate not only the remote tele-automaton, but also portions of the remote 
environment itself. Forward simulation will work fine when interacting with static objects, but what 
about interactions with moving objects? The simulation based methods we have discussed are depen- 
dent, in pure form, entirely upon the quality of the robot and environment models available and the 
accuracy with which tasks must be performed. In all of our experimental tests to date, the accuracy 
required was well below the accuracy of the models, and this was not a problem. However, most 
assembly tasks involve contact among the parts and have much higher accuracy requirements. More- 
over, independent of accuracy requirements, even small errors when contacts are involved can produce 
very high, possibly damaging, forces. Solutions for this important class of problems is essential for 
many, if not most, applications. [VOL88] describes these basic problems in greater detail and outlines 
a number of possible directions for solution. 

Further work is needed on methods for path-error specification and associated methods for the time 
optimization of path following, such as in [SUH87]. Additional work is also needed on autonomous 
“reflex” actions that the remote robot can perform when encountering uncertainties (particularly 
those involving contact) not modeled in the forward simulation. We also see the need for augmented 
AI programming environments that interface in such a way with real-time programming environments 
as to easily enable rapid estimation of time available for short-term AI planning tasks (enabling us 
to select among AI methods as a function of available time). 

Exploration of different dimensions of tele-autonomy is also likely to lead to near term advances 
of considerable utility. In particular, most of the base technologies exist for developing what we have 
called “remote coaching” systems in which a local expert can coach a remote technician in complex 
experimental or maintenance tasks. A prototype remote coaching system is described in [WAL88]. 
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This prototype includes a modest, but extensible, expert system in the remote controller that can help 
the technician with most problems and call in the expert when needed. The system allows graphic 
or image data objects to be selected from a library and placed on workstation screens for both the 
technician and the expert. The expert has a graphic capability for drawing on both screens, as well 
as being in voice contact with the technician. Even slow scan video is available. Work is still needed 
on the most effective means of on-line interaction, however, such as simulation video and graphics, 
and the extension of collaboration technology to the domain of cooperative manipulation tasks. 


5 Conclusions 

We have pointed out that there is a growing need in many areas of our society to be able to 
achieve remote intelligent action at a distance, and that traditional methods of automation and 
artificial intelligence are inadequate for such tasks. We have further introduced three dimensions that 
characterize the problem space: (i) the type of process being performed (perception, cognition, and/or 
action), (ii) the form that is performing the process (human, robot, automatic vehicle, etc.), and (iii) 
the location at which the process is being performed. We have coined the term tele-autonomous 
systems to describe systems addressing this problem space. Tele-autonomous systems are represented 
by a set of points in this space spread across more than one location plane. 

One of the most fundamental problems that must be overcome in building such tele-autonomous 
systems is time delay resulting from telemetry or signal propagation. Simulation of remote devices 
and environments is part of the solution. We have introduced the notions of time and position 
de-synchronization (implemented through time and position clutches) to allow the simulation to be 
operated faster than real time and to permit an on-line “motion editing” to be achieved. Our early 
experiments involving the use of such time and position clutches suggest that dramatic improvements 
in performance can be achieved through the use of these clutches, even when there is no time delay. 
Moreover, the time and position clutches can be used to accomplish a new interaction protocol for 
hand-offs between two agents controlling a remote device. This protocol is based upon a shared 
visualization of the intended motion of the device. 

While the number of potential applications for tele-autonomous systems is immense, there is yet 
a great deal of research to be done. We concluded by identifying just a few of the areas needing 
research. Among the more important were the extension of the time and position clutch ideas to 
situations involving precision contact among the objects involved, the development of “remote coach- 
ing” systems, learning systems, and the development of collaboration technology to support group 
manipulation tasks. 

We believe that tele-autonomous systems research can yield methods and systems for improved 
projection of intelligent action at a distance in time and space. This interdiscipline presents inter- 
esting new research opportunities to teams having expertise in robotics and automation, artificial 
intelligence, and the psychology of human-computer interaction. We envision many possible applica- 
tions for the resulting technology, not only in space and defense systems, but also in design systems, 
production systems, and eventually in personal and recreational environments. 
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Figure 1: Dimensions of the problem space of Figure 2: Cooperative repair of a satellite by 
intelligent action at a distance. mission specialist in the Space Station and ex- 

perts on Earth. 



Figure 3: Basic components of a tele- Figure 4: Using forward simulation and pre- 

autonomous system. dictor display to cope with time delay (after 

Noyes and Shridan [NOY84]). 




Figure 5: Visualizing manipulation through a Figure 6: Task completion time as a func- 
time delay using forward simulation. tion of task difficulty and communication de- 

lay, showing performance improvement using 
forward simulation. 
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Figure 7: Rapid manipulation path gen- Figure 8: Initial trail results, showing Ts,Tm 

eration using forward simulation with time as functions of system and task parameters for 
clutch. three models of control. 



Figure 9: Using the position clutch to cope 
with a more difficult manipulation. 





Figure 10: Using time brake to handle a con- 
tingency. 



Figure 11: Application of manipulation task Figure 12: Using time and position clutches 
handoff through a time delay, using tele- to handoff task to another forward simulation 
autonomous system controls. agent. 


157 








